[TEST] Golden file test infrastructure for JSON Schema baselines#471
Open
Seth Fitzsimmons (sethfitz) wants to merge 1 commit intodevfrom
Open
[TEST] Golden file test infrastructure for JSON Schema baselines#471Seth Fitzsimmons (sethfitz) wants to merge 1 commit intodevfrom
Seth Fitzsimmons (sethfitz) wants to merge 1 commit intodevfrom
Conversation
Helpers `assert_golden` and `assert_json_schema_golden` live in `overture.schema.system.testing`. The pytest plugin at `overture.schema.system.testing.plugin` registers `--update-baselines` and the `update_baselines` fixture; consuming packages opt in via `[project.entry-points.pytest11]`. Helpers live in `system` rather than a separate test-support package: `system` is foundational, helpers depend on `system.json_schema`, and themes already depend on `system`. Mirrors `numpy.testing` / `pandas.testing`. Per-package opt-in rather than a single entry point in `system`: keeps `--update-baselines` from appearing in `core`, `codegen`, or `cli` test runs when consumed in isolation. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
b133862 to
215756f
Compare
Collaborator
Author
|
Victor Schappert (@vcschapp) I re-did this PR in response to our conversation a few weeks back. Ready to look at again. |
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces hand-rolled JSON Schema baseline tests with a shared pytest plugin. Each per-feature test drops from ~35 lines of fixture boilerplate to ~10 lines of declarative call.
What's new
overture.schema.system.testing(inoverture-schema-system):assert_golden(actual, golden_path, *, update)-- compare a string against a golden file; on mismatch, raisesAssertionErrorwith a unified diff. Withupdate=True, writes instead of comparing.assert_json_schema_golden(model_or_union, golden_path, *, update)-- generate the JSON Schema for a Pydantic model (or discriminated union) and delegate toassert_golden.overture.schema.system.testing.plugin-- pytest plugin that registers the--update-baselinesflag and theupdate_baselinesfixture.Helpers live in
system(not a separate test-support package) becausesystemis foundational, the helpers depend onsystem.json_schema, and themes already depend onsystem. Mirrorsnumpy.testing/pandas.testing.Per-package opt-in
The plugin is activated by each consuming package's
pyproject.toml:This keeps
--update-baselinesfrom appearing incore,codegen, orclitest runs when those packages are consumed in isolation.Each baseline test is marked
@pytest.mark.baselineso they can be selected or skipped (pytest -m baseline/pytest -m "not baseline"); the marker is registered per-package in[tool.pytest.ini_options].Test simplification
All 16 per-feature baseline tests reduce to:
Other changes
Makefile:reset-baseline-schemas(which deleted files for regeneration on next run) replaced withupdate-baselines, which runspytest --update-baselines -m baseline.pyproject.toml: registers thebaselinemarker at the workspace level for selective runs.packages/overture-schema-system/README.md: documents the helpers, plugin opt-in, and update workflow.packages/overture-schema-system/tests/testing/test_golden.py: tests for the helpers themselves.Net: ~435 lines removed, ~416 added (mostly the new helpers, plugin, README section, and their tests); per-test files shrink substantially.
Closes #470